feat(cloudflare): add optional R2 batch uploads via rclone for cache population #925

krzysztof-palka-monogo · 2025-10-03T14:48:10Z

Batch upload R2 cache using rclone

This PR proposes adding support for optional batch uploading R2 cache using rclone.
Based on the solution discussed in Issue #866

Details

Automatic selection between batch vs standard upload

The CLI now detects when all of R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, and CLOUDFLARE_ACCOUNT_ID are set in the environment and switches to batch upload mode. Otherwise, it falls back to the standard (Wrangler-based) upload method.

Graceful fallback on errors

If batch upload fails (e.g. due to rclone errors), the flow will log a warning and transparently revert to the default upload approach.

Staging and parallel transfers

Assets are copied into a temporary staging directory, then uploaded using rclone copy with concurrency settings (--transfers=32, --checkers=16) for performance.

Related issues

changeset-bot · 2025-10-03T14:48:14Z

🦋 Changeset detected

Latest commit: fe8f553

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@opennextjs/cloudflare	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vicb · 2025-10-03T17:22:10Z

@krzysztof-palka-monogo thank you so much for this PR, it will please many of our users!

I only took a quick look and I'll do a full review on Monday.

I have some preliminary comments/thoughts:

I think we should rename --rcloneBatch. First we do not want to expose the implementation (rclone) as it might change - we can keep mention to rclone and in the doc. Ideally we can re-use cacheChunkSize
rclone.js was last released 3y ago - Ideally we could find a more recent and maintained implementation (but we should not block the PR on that, it could be a follow-up or we can wait until we actually have an issue)
We should info log when users are not using batch upload and have i.e. > 10 assets. Another option could be to batch upload by default and fallback to slow/safe on error (again happy if this addressed as a follow up)

krzysztof-palka-monogo · 2025-10-06T06:22:19Z

Good morning @vicb

Here’s the justification for keeping an explicit --rcloneBatch (or similarly named) flag and retaining the existing upload mechanism as the default:

Requires external configuration
Rclone is not a drop-in replacement — it depends on a separate config file (the rclone config, credentials, remotes, etc). For users who have never used rclone, they must explicitly create and supply that config. That means it is not “just work” out of the box.
Avoids a breaking change
If we made rclone-based behavior the default, we risk breaking setups for users who do not have configured rclone. Keeping the current behavior as default ensures backward compatibility.
Explicit intent
Having a named flag (e.g. --rcloneBatch) clearly signals the use of rclone and keeps any implementation coupling explicit (rather than hidden). It also gives future flexibility: if one day we swap in a different batch uploader (or abstraction), the explicit flag boundary helps isolate that change.

vicb · 2025-10-06T06:33:53Z

Here’s the justification for keeping an explicit --rcloneBatch (or similarly named) flag and retaining the existing upload mechanism as the default:

We definitely don't want "rclone" in the flag. We could use another mechanism, i.e. remote bindings.

Requires external configuration
Rclone is not a drop-in replacement — it depends on a separate config file (the rclone config, credentials, remotes, etc). For users who have never used rclone, they must explicitly create and supply that config. That means it is not “just work” out of the box.

IMO we should drop the need for an external config in favor of using env vars - the CLI could dynamically create a temp config file for rclone.js as it doesn't seem to support env vars.

Avoids a breaking change
If we made rclone-based behavior the default, we risk breaking setups for users who do not have configured rclone. Keeping the current behavior as default ensures backward compatibility.

We can use rclone when configured and fallback to the current mechanism - but log that there is faster way and link some docs.

Explicit intent
Having a named flag (e.g. --rcloneBatch) clearly signals the use of rclone and keeps any implementation coupling explicit (rather than hidden). It also gives future flexibility: if one day we swap in a different batch uploader (or abstraction), the explicit flag boundary helps isolate that change.

I disagree here.
Users want something that works (fast), they should not have to care about how it is implemented.

… variables

krzysztof-palka-monogo · 2025-10-06T08:59:36Z

Hi @vicb

I've updated the PR to align with your suggestions:

Removed --rcloneBatch flag and all rclone references from user-facing API
Implemented environment variable configuration (R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_ACCOUNT_ID)
Batch upload now activates automatically when credentials are detected
Fallback to wrangler uploads when credentials aren't provided
Temp config files are created dynamically with secure permissions and
auto-cleanup

Documentation now focuses on "batch upload" as an optional performance feature, not implementation details.

Please let me know if thats will meet the requirements and if so could we run "Publish prereleases" action to properly test the changes in some real applications 😄

pkg-pr-new · 2025-10-06T09:46:16Z

Open in StackBlitz

npm i https://pkg.pr.new/@opennextjs/cloudflare@925

commit: 0d53eac

krzysztof-palka-monogo · 2025-10-06T12:50:18Z

Hi @vicb

I will be glad for code review.

.changeset/rclone-batch-upload.md

vicb · 2025-10-06T13:39:52Z

packages/cloudflare/src/cli/commands/populate-cache.ts

+function createTempRcloneConfig(): string | null {
+	const accessKey = process.env.R2_ACCESS_KEY_ID;
+	const secretKey = process.env.R2_SECRET_ACCESS_KEY;
+	const accountId = process.env.R2_ACCOUNT_ID;


To avoid double access, maybe there could be a retrieveR2CredentialsFromEnv returning {R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, CLOUDFLARE_ACCOUNT_ID} (I don't think the account as anything to do with R2)

vicb

Note sure I'll have time for a full review today, sorry.

A couple comments:

R2_ACCOUNT_ID should rather be CLOUDFLARE_ACCOUNT_ID as it is not specific to R2
Thanks for implementing the env vars based solution for CI. I'm wondering if we can improve the story for local dev. It would be nice to get the vars from .env / .dev.vars files but it would mean adding vars that are only used in local dev and that might confuse users. Maybe we could look for a local <pjt>/rclone.conf and fallback to env vars when it's not here. What do you think?

…UDFLARE_ACCOUNT_ID

krzysztof-palka-monogo · 2025-10-07T07:13:05Z

Hello @vicb

I've addressed both points:

Renamed R2_ACCOUNT_ID to CLOUDFLARE_ACCOUNT_ID
The CLI now loads environment variables from a .env file in the project root.

This keeps the implementation simple while making local dev more convenient. WDYT?

vicb · 2025-10-07T08:22:00Z

packages/cloudflare/src/cli/commands/populate-cache.ts

+		try {
+			// Cleanup temporary config file
+			rmSync(tempConfigPath);
+		} catch {
+			console.warn(`Failed to remove temporary config at ${tempConfigPath}`);
+		}


maybe use force and recursive instead of try/catch?

vicb · 2025-10-07T08:25:44Z

This keeps the implementation simple while making local dev more convenient. WDYT?

I don't think there is an ideal solution for .env vs a local config file.

That being said, Next.js uses .env files with this loading order so we should make sure we are consistent.

I'll take a deeper look today and comment on how we could do that

krzysztof-palka-monogo · 2025-10-07T08:41:55Z

I’ll be away for a week starting tomorrow, so I need to wrap up this functionality today 😅
Since our ideas have been evolving, could we prioritize only essential changes now and handle the rest in a follow-up iteration?

feat: add --rcloneBatch flag for faster R2 cache uploads using rclone

d8171c5

krzysztof-palka-monogo added 2 commits October 3, 2025 16:54

fix(test): remove useless test

5aefc21

test: add tests for populateCache with R2 cache

c79d18d

krzysztof-palka-monogo added 3 commits October 6, 2025 08:37

commit to trigger pkg-pr-new action

d0c5606

refactor: not expose rclone to user and use batch upload based on env…

15a874f

… variables

test: update populateCache tests after implementation changes

0d53eac

krzysztof-palka-monogo changed the title ~~feat(cloudflare): add --rcloneBatch flag for faster R2 cache uploads using rclone~~ feat(cloudflare): add optional R2 batch uploads via rclone for cache population Oct 6, 2025

fix: enhance error handling in populateCache for rclone uploads

3d181e5

vicb reviewed Oct 6, 2025

View reviewed changes

.changeset/rclone-batch-upload.md Show resolved Hide resolved

docs: add links for creating R2 access API tokens in README

32dc294

vicb reviewed Oct 6, 2025

View reviewed changes

krzysztof-palka-monogo added 2 commits October 7, 2025 08:27

fix: update environment variable references from R2_ACCOUNT_ID to CLO…

a473d6e

…UDFLARE_ACCOUNT_ID

feat: add support for loading environment variables from .env file

fe8f553

vicb reviewed Oct 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cloudflare): add optional R2 batch uploads via rclone for cache population #925

feat(cloudflare): add optional R2 batch uploads via rclone for cache population #925

krzysztof-palka-monogo commented Oct 3, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Oct 3, 2025 •

edited

Loading

Uh oh!

vicb commented Oct 3, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025

Uh oh!

vicb commented Oct 6, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Oct 6, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025

Uh oh!

Uh oh!

vicb Oct 6, 2025

Uh oh!

vicb left a comment

Uh oh!

krzysztof-palka-monogo commented Oct 7, 2025

Uh oh!

vicb Oct 7, 2025

Uh oh!

vicb commented Oct 7, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 7, 2025

Uh oh!

Uh oh!

feat(cloudflare): add optional R2 batch uploads via rclone for cache population #925

Are you sure you want to change the base?

feat(cloudflare): add optional R2 batch uploads via rclone for cache population #925

Conversation

krzysztof-palka-monogo commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Batch upload R2 cache using rclone

Details

Automatic selection between batch vs standard upload

Graceful fallback on errors

Staging and parallel transfers

Related issues

Uh oh!

changeset-bot bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vicb commented Oct 3, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025

Uh oh!

vicb commented Oct 6, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Oct 6, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 6, 2025

Uh oh!

Uh oh!

vicb Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

vicb left a comment

Choose a reason for hiding this comment

Uh oh!

krzysztof-palka-monogo commented Oct 7, 2025

Uh oh!

vicb Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

vicb commented Oct 7, 2025

Uh oh!

krzysztof-palka-monogo commented Oct 7, 2025

Uh oh!

Uh oh!

krzysztof-palka-monogo commented Oct 3, 2025 •

edited

Loading

changeset-bot bot commented Oct 3, 2025 •

edited

Loading

krzysztof-palka-monogo commented Oct 6, 2025 •

edited

Loading